Anne Paxton
September 2020—Bill Gates was just 10 years old and the Beatles were still playing live concerts when the concept of patient-based real-time quality control was proposed in 1965.
At the time, patient-based real-time QC (PBRTQC) was based on the “average of normals,” a precursor of moving averages. And outside of hematology, where PBRTQC has been widely used for 40 years, conventional QC has continued to reign. But recent advances in data analytics and computing power could now allow more laboratories to sidestep the downsides of conventional QC materials by using a computer to perform continuous QC on clinical chemistry tests.
PBRTQC got a stamp of approval last year when a working group of the International Federation of Clinical Chemistry and Laboratory Medicine (IFCC) Committee on Analytical Quality recommended increased adoption of PBRTQC techniques to improve the reliability of clinical laboratory test results (Badrick T, et al. Clin Chem. 2019;65[8]:962–971). “The recent successful implementation and proof of value of patient-based real-time QC in complex labs has further given confidence in this technique,” says IFCC working group member Mark A. Cervinski, PhD, medical director of clinical chemistry and director of point-of-care testing at Dartmouth-Hitchcock Medical Center, which in 2013 adopted moving average protocols in the laboratory.
Large and small labs will realize the benefits of using PBRTQC, says IFCC working group member Alex Katayev, MD, director of clinical science assessment at LabCorp. “Obviously, some learning is required, but there’s a lot of information about real-time quality control out there now. It’s completely different from what was there five years ago.” PBRTQC will be the future of laboratory industry QC, in his view. “It’s just a matter of time,” he says.
Moving averages, commonly used with time series data to smooth out short-term fluctuations and highlight longer-term trends, have become better known of late because they are referenced daily in press reports tracking the spread of COVID-19 cases and deaths. In laboratory medicine specifically, moving averages are continuously updated averages of test results that can be compared continuously in real-time to normal test results.
One of the clearest benefits of using moving averages to conduct PBRTQC is the effect on an important number: ANP, the average number of patient samples affected from onset of error to detection. In part by reducing ANP, PBRTQC can slash the number of incorrect laboratory results before they are reported, as well as the number and cost of retests.
PBRTQC is more difficult to implement than conventional QC because it requires accessing patient data, setting up appropriate rules and action protocols, and choosing the best statistical algorithms. But its many virtues make it worth pursuing, say the IFCC recommendations. Traditional QC, which is retrospective, not real time, and can detect error only at a subsequent measurement of QC sample, may be performed only once or twice a day. It may also use noncommutable materials and fail to detect small positive biases, major analytical shifts, long-term analytical drift, and imprecision at low but clinically important concentrations.
By contrast, PBRTQC is risk-based, clinically relevant, and cost-efficient. There is no additional cost to performing PBRTQC once the system has been set up, and it can be used for uncommon assays for which conventional QC materials may be hard to find. Perhaps most significant, as a 2014 study by Dr. Katayev demonstrated, moving average algorithms can effectively keep most erroneous results from going out the door (Fleming JK, et al. Clin Biochem. 2015;48[7–8]:508–513).
As director of QC at LabCorp, Dr. Katayev has a unique lens on real-time quality control and its rich potential benefits. LabCorp started in about 2011 to plan modern, more sophisticated replacements for liquid or lyophilized quality control because of mounting evidence that conventional QC materials have commutability problems and sometimes do not behave the way patient samples behave, owing to matrix effects from the preservatives the QC materials contain. “We run a lot of volume, and with that volume you obviously have more incidents of those misbehaviors in QC,” Dr. Katayev says.
LabCorp’s size meant the company had larger block sizes, the number of patients that can be analyzed by middleware for QC as a single point, allowing a “report of the back” approach to issuing results, he explains. Working with middleware provider Data Innovations, by 2013 LabCorp had started to implement PBRTQC gradually in its regional laboratories. Today, “We do not report any results until the entire block, typically 50 patient results, is processed, and the first result is reported only if all rules for the given block are passed. So basically not a single patient result is reported that may be compromised.”
Smaller laboratories can use different weighted algorithms with smaller block sizes, he says. “There is some risk of reporting some compromised results, but because the labs are small and typically in a hospital environment, they can react to this relatively fast. And it is still a much faster reaction time than if you used conventional LQC in the middle of the run or end of the run. So it’s highly unlikely that the patient’s management would be compromised by an incorrect result.”
Dr. Katayev and other members of the IFCC working group have helped develop recommendations for commercial simulation software available from European vendors that any laboratory can use. “It’s not free, but it’s not that expensive, so basically any hospital laboratory can buy the license and use it,” he says. “You don’t need to reinvent the bicycle, so to speak.”
[dropcap]I[/dropcap]mplementing PBRTQC always includes three steps, Dr. Katayev explains. “You need to first determine your analytes that will be subject to this control, your volumes, your laboratory population, whether it’s hospital or outpatient. Some laboratories have both outpatients and inpatients and may need to use different rules for those patients.”
The second step is to use a simulation to develop rules. “You just run your results from your laboratory information system through the middleware and test your algorithms and see what is the best. We did it pretty much by trial and error back in 2012. But now you can apply some help from available software that will help you pick the best algorithm, block size, thresholds, error limits, filtration, and so forth.”
Finally, the simulation must be validated with real results. “We grab patient results again and run them through the software with applied rules and see how the results were affected. Then we inject artificial biases or errors in the patient results to see how they would detect an out-of-control situation.”
The end product has been a system that generates analysis of chosen quality rules for the moving block of patient results with alarms in real time that can be detected by staff immediately. “We have, number one, a visual alarm. Each instrument has a light bulb, so if it’s ever flashing red, that means something is wrong,” Dr. Katayev says. “And some workers also set computer notifications so the manager would see on his computer screen right away if something is going wrong.” Automatic phone calls or text messages also are possible. “Some labs use audio alarms, but I think audio alarms may cause alarm fatigue. So visual works just fine.”
The obstacle to widespread use of PBRTQC, until recently, has been the amount of computing power needed to make it work, he says. “You need very powerful servers. You need to move huge amounts of data between instruments, laboratories, and servers. And the data need to be processed relatively fast in real time.”
“Unfortunately, the tools that many labs have in place right now are not compatible with PBRTQC. But I know that for laboratories of our size, it’s very cost-effective. Even with the initial investment in the Data Innovations middleware, the savings we are experiencing through implementation are huge”—on the order of millions of dollars a year, he says.
How were those savings gained? “Number one, we’ve cut the use of liquid QC material by 75 or 80 percent. So that’s huge because we buy a lot and it is not cheap. Number two, we save a lot of labor and reagent cost because we’ve cut our repeat testing rate by about 50 percent.”
For clinical chemists and laboratory directors who are not deeply involved in PBRTQC, it is an entirely new area, Dr. Katayev says. “But we need to understand that the old way of doing QC, using parametric rules, is not scientifically sound. Because all those rules are based on the assumption that the distribution of QC materials is parametric or Gaussian, or near Gaussian. But most of the distribution of QC materials in chemistry is not Gaussian. It is skewed.”
The consolidation of laboratories has helped to bolster the possibilities for implementing PBRTQC, he says, as has the increased harmonization across manufacturers. “You can pretty much obtain the same result from different methods for the same patient. That means you can create a network between laboratories and you can even increase the testing sites, so to speak, by networking between laboratories and monitoring them all through these patient-based real-time QC algorithms. With a dashboard of these laboratories, at LabCorp, you can proactively see the shifts, even remotely, then call the laboratory and warn them before a potential disaster can occur.”
Implementing moving averages does require a certain minimum test volume. “If your test volume is very, very small, there is no way you can use it and no need to use it,” Dr. Katayev says. Similarly, financial constraints may make PBRTQC too costly for smaller laboratories. “The biggest constraint, though, is a change in mentality and learning. When we implemented it, the first year or so people were reluctant and skeptical. There was a lot of pushback, as always happens when you do something new.”
In time they created standard operating procedures that described every step and what to do for whatever might happen. “We implemented all those SOPs in the lab and trained, trained, trained and learned, learned, learned. Then people started loving this thing because they realized the benefits. If you look at the number of corrected reports—and those are a nightmare, right?—in our labs those have gone down significantly, because we do not report any single result without having passed the entire set of QC rules.”
Hospital labs often don’t have the luxury of waiting for an entire set of QC rules to pass because they need results to be reported immediately. “But, again, in a hospital environment they will still be able to see the shape of the QC graph, and even if they reported that result 10 minutes ago they can call the ward immediately and say, ‘Hey, that one is off.’ Because they can still detect it much faster than with conventional QC.”
[dropcap]P[/dropcap]BRTQC works best for higher-volume chemistry tests like sodium, potassium, calcium, creatinine, and hemoglobin A1c, Dr. Cervinski says. “It doesn’t work as well for tests, such as glucose, that are highly skewed between people. When some patients come in with a glucose of 400 and others have a glucose of 50, you can’t really do an average with that.”
In hematology, laboratories have long used a moving average called Bull’s algorithm, he says. “I think part of the reason why real-time QC became more accepted in hematology is at the time hematology did not have good, stable QC materials, whereas in the chemistry lab acceptable QC materials were available.” Also, historically, the computers needed to pull off the calculations needed in real time did not exist at a reasonable cost until 10 years ago. “You had to have a supercomputer 20 years ago to do this and it wouldn’t be very feasible. But now the processing power of your typical desktop computer is phenomenal.”
Dartmouth-Hitchcock Medical Center, the only level one trauma center in New Hampshire, includes a 400-bed hospital; with several physician office practices and large and small clinics, the medical center performs about 4.5 million chemistry tests a year. “But as the test menu and number of daily runs grew, the number of QCs we were running each day didn’t grow at the same rate,” Dr. Cervinski says. “Even if you’re running conventional QC every two hours, you can still have several hundred patients affected by an error before you’d ever detect it. And that’s assuming you detect it on the first QC event after that error initially occurred.”
Moving averages reduce the time to detect errors by continuously updating the set of samples being averaged, he says. “If you are averaging 10 patients, then you analyze the 11th patient and the oldest patient value drops out of the population. This is an improvement on the older technique of an ‘average of normals,’ which would compare, say, the first block of 20 patient results’ average with the second block of 20 results’ average, and so on.”
Thinking of errors as persistent events can be a mistake, Dr. Cervinski points out. “We typically think that something fails and stays failed until we intervene. But that’s perhaps a naive assumption. You can have analytic errors that happen intermittently that will be there for 10 or 30 or 40 minutes and then self-resolve. You would never catch those with conventional QC materials.” So while PBRTQC is not a perfect solution for every analyte or every laboratory, “when it’s applied to a proper test it can be quite beneficial.”
Real-time quality control may involve near-continuous rather than strictly continuous real-time QC. Calcium is a good example, Dr. Cervinski says. “We monitor the average calcium for all patients but we also subdivide them for inpatients only and outpatients only because those averages are not the same. Similarly, we’ve taken the extra step here of excluding certain populations based on disease status. For populations that tend to be far more ill, their laboratory values tend to deviate much further from the mean.”
[dropcap]U[/dropcap]sing patient data in real time, not just collecting and looking at it, can be the big challenge with PBRTQC. Like many others familiar with and enthusiastic about the benefits of PBRTQC, Clayton R. Wilburn, MD, medical director of clinical chemistry, immunology, and point-of-care testing at the University of Vermont Medical Center, has Data Innovations middleware and would like to make the jump to real-time QC but has encountered obstacles.
DI middleware does have supportive features for PBRTQC, including modules for moving averages. It’s a middleware solution for many laboratory platforms between the instrument and laboratory information system, says Dr. Wilburn, a member of the CAP Clinical Chemistry Committee.
Part of the process of evaluating a failed quality control issue for an analyte is correcting prior laboratory reports. “When we do have QC issues, I pull back samples from patients that were released between the last approved QC and the retest. And sometimes I find that the results are significantly different.” Ideally, “a real-time system would prevent that from happening, or at least lessen the frequency.”
Dr. Wilburn’s own experience with PBRTQC was during a fellowship when his program began to do PBRTQC with calcium. “It was relatively stable and you could see the data for our populations showed it would not fluctuate very much.” Labs with patient populations that are well defined and don’t fluctuate considerably—say, by people visiting in the summer or winter months—would probably be good candidates for benefiting from PBRTQC, he says.
To varying degrees, analyzer software, middleware, or laboratory information systems can all support PBRTQC. But the cost of implementing PBRTQC, Dr. Wilburn has found, is a variable. “Lots of people use DI for their middleware so it’s just an add-on to that. But one of the bigger things to take into account is the IT support you’ll need as well as your own comfort level as a medical director and how knowledgeable you are in terms of computation and if you have any programming experience.”
“The folks that do really well with it can engineer it from scratch. They often have programming experience and are fairly comfortable with the informatics pipelines and how data are evaluated in those systems, versus something that is more ‘shake and bake’”—that is, more of a completely built system. You might say: “‘I want X, Y, and Z analyte to be used and I want to look at it this way,’ and then the system does it for you. It comes down to how applicable the program is and how much individual customization and understanding are needed in terms of programming and the analytics.”
Getting the system up and in working order can be a larger lift for people without experience in big data analysis, Dr. Wilburn says, noting that his own laboratory is still probably several years away from implementing PBRTQC.
In flat dollars and cents, QC is not cheap, he points out. “Especially if you’re a 24/7 operation and you’re running it multiple times a day, traditional QC starts eating up quite a bit of your budget. If you are basically using your patient samples, which you are already running, then you can lower the cost of those reagents. You’ll need the basic calculation of how many years of using this system with a reduction of QC it would take to break even for you; then you’ll need to present it to your institution or administration and ask, ‘Is this something you would find to be worthwhile?’ Because it’s both trying to save money and trying to improve results that you’re putting out for patients.”
PBRTQC used over the long term can reduce conventional QC use and lead to appreciable cost savings, Dr. Wilburn says. “How often do you encounter a situation where there was a failure of the QC, you didn’t detect a shift, and a value went out and you have to worry there was appreciable harm to the patient? Most of the time, there’s not.” But, he asks, what value would you ascribe to PBRTQC if it could prevent such outcomes?
Answering the question of which type of assays would be most suitable means asking, “How much data can you take?” Dr. Wilburn believes. “As you add more and more, of course, there are more and more data points to consider in terms of storage space and more computational power you need to take into account.”
But he has found PBRTQC’s usefulness undeniable. “If there is an analyte you are having issues with and you see that you’re calibrating it more frequently than the manufacturer would recommend, and it’s not an issue with your instrument, these might be assays where you would find it useful to detect these minor shifts before they start to lead to major shifts causing you to have to go back and retest specimens.”
The size of the laboratory could determine whether PBRTQC is practical. But Dr. Wilburn thinks it comes down to the time commitment of the people available to work on a PBRTQC system. “Maybe using PBRTQC on selected analytes on a smaller system in a smaller-scale laboratory would make sense. But there are pre-made solutions and also homemade solutions by people who may have some R programming language experience. It’s going to depend on the comfortableness and experience of the medical director or the IT or informatics folks and what they can put in timewise” for an in-house solution, “versus installing a system bought straight from a company.”
A common roadblock can be institutional infrastructure—space for the server and sufficient computational power. But humans can pose roadblocks, too, because of unfamiliarity with computing, Dr. Wilburn says. “If you go to someone who doesn’t have a lot of informatics experience and the options for patient-based QC will require more intricate knowledge of informatics in terms of how data flows, how calculations are made, how often data is pinged on the server, that may well be out of their wheelhouse.”
PBRTQC can cause frequent interruptions in the laboratory because testing stops when a quality problem is sensed. Does this make PBRTQC difficult to manage logistically? It could, Dr. Wilburn says. “If you set the whistle bell at a really low threshold, it would be interrupting the testing cycles more frequently than with every-12-hour or 24-hour QC. So you have to tailor the process to what level of risk you’re willing to have.” The size of the laboratory could influence that tailoring. “In a small laboratory that has a single instrument and doesn’t have a lot of duplication in terms of its capabilities, you don’t want to constantly bring an instrument down because of trying to chase your tail on small problems that may end up not being worthwhile.”
The ideal answer to that, Dr. Wilburn believes, would be a PBRTQC system that is artificial-intelligence–based, where the system performs its monitoring function and requires less assistance in deciding what’s there or not there. But that is likely several years away, he says.
PBRTQC can sound daunting, Dr. Wilburn admits, and he advises laboratories that are interested in it to start with a general moving average on a few analytes. “If you can understand the underlying principles and get comfortable with talking about how the data is looked at, you can then scale that up. There’s not going to be a one-size-fits-all. It depends on your lab, your comfort level, and your patient population. But realize there are resources out there. Some colleagues have already been doing this for several years. And don’t be afraid to ask questions and admit you need help with it.”
[dropcap]B[/dropcap]renda Suh-Lailam, PhD, director of clinical chemistry and point-of-care testing at Lurie Children’s Hospital of Chicago, has found that, even with adequate test volume, taking advantage of PBRTQC can prove to be an elusive goal. Her lab, which has Data Innovations middleware for its Roche instruments, began about four years ago with two tests, sodium and potassium, to see if a more comprehensive use of PBRTQC would be practical.
At that time, the laboratory encountered difficulties with the real-time aspect of the QC program it had deployed. When moving averages were drifting outside limits, for example, “it could send email alerts to a certain person, signal high or low, and hold patient results for investigation before reporting, but we were unable to get historical data for further investigation and comparison as the server size was not big enough to store the large amount of data being generated. So we had to regularly store screenshots of the moving average graphs and data tables. And we did not have dedicated staff to perform these workarounds,” says Dr. Suh-Lailam, a member of the CAP Clinical Chemistry Committee.
The hospital’s upgrade to a newer version of Windows, which was not compatible with the Data Innovations version in use, brought the moving averages function to a standstill, so she was compelled to disable that function. “We worked with Data Innovations to figure out how to make the middleware compatible. But we knew it was going to be a problem every year or two whenever we would upgrade Windows. We wanted something that was sustainable, and we had limited resources.”
Because the Lurie Children’s pathology department has worked to build capacity for data analytics, her lab was able to take an alternative approach. Data from Epic, the LIS, and other hospital systems were integrated and a dashboard was created to display a time series of aggregated patient results with frequent updates, all using the hospital’s licensing for Microsoft’s Power BI tool. “We can log in and see: Are we in line? Do we have any spikes, dips, or drifts? Is the average moving outside desired thresholds? The integration with Epic and other data also facilitates investigations into concerning results or trends.”
At this time, they have not built in alerts, much as the lab would like to, and the system does not have a feedback mechanism to the instruments or middleware, Dr. Suh-Lailam says. “So it’s user-driven to provide us with frequent snapshots and is not fully integrated with the instruments. It’s an evolving system. The moving averages component is only one aspect of the overall dashboard. We’ve been able to build a lot into it, even though it’s not fully functional as a PBRTQC system. But the advantage it has over DI is that to use Data Innovations for PBRTQC, we would have to go in and rebuild for every single test.”
Thus, her laboratory is not yet at the point at which it can replace traditional QC. “It could be more feasible down the line. Because we never had these real-time interactive dashboards before, and more data analytics tools are becoming available and more laboratorians are becoming knowledgeable on how to use data analytics in laboratory medicine, it’s possible that in the future more of us could be doing patient-based QC.”
She believes in the concept of PBRTQC because conventional QC is run only intermittently and periodically and a lot of patient results could be reported before the laboratory realizes something went wrong. “We have a vice chair of computational pathology who is proactive in working with the hospital to build data analytic tools for the laboratory, including those relying on Power BI. And I think labs need to be proactive and make it clear that it’s very important to have the ability to look at patient moving averages in real time.”
Dr. Suh-Lailam’s experience is not unique. As other members of the CAP Clinical Chemistry Committee have pointed out, even for labs with adequate volume, PBRTQC is harder to do than people might think.
That’s a fair statement, Dr. Cervinski says. “Anytime I talk about this, I fully admit it’s one of the things that’s preventing PBRTQC from being in every hospital laboratory.” As of yet no commercial tools are available to generate optimal moving averages settings, he says. “With the IFCC working group, we have been trying to push for the middleware manufacturers to come up with an out-of-the-box solution. We realize that one of the major limitations to wider adoption of patient-based real-time QC is that you have to spend a significant amount of time to set it up in order for it to work well.”
Dr. Cervinski thinks Bio-Rad Laboratories and other middleware vendors could develop such a solution. “Or one of the large multinational diagnostic companies with their own IT group, if they put their minds together. With the right business model, they could develop a solution and market it.”
Huvaros is another option. A website developed by Dutch clinical chemist Huub van Rossum, PhD, Huvaros allows users to put in deidentified data and come up with different parameters, Dr. Cervinski explains. “It will tell you how many samples, on average, it would take to detect that error. Then it gives you a couple of different options.” He has seen a demonstration of Huvaros and believes it can generate far better information than most people could generate on their own. Programmers could develop a user-defined software using the languages Python or R, he adds. At Dartmouth, “we did all our work using MATLAB, which is basically a mathematical modeling software.”
The moving averages system he set up at Dartmouth was a joint project with Data Innovations and a resident who had the software and scriptwriting skills to develop a script that could be used to optimize averaging parameters. “And one of our LIS staff has been very helpful in debugging some of the quirks in the software that we wouldn’t be able to take care of alone.”
Though the software has been affordable, Dr. Cervinski knows that would not be true everywhere. And he’s not surprised to hear about Dr. Suh-Lailam having to disable her real-time QC program. “Setting up moving averages is a serious time commitment,” he says, “and I am not doing this on my own.” His LIS team is key to keeping the software running with the various system updates. “The reality is, if it is not set up correctly, you may miss significant events or have a lot of false results. That’s one of the things I caution people about. And quite honestly, when we first set our system up, I was disappointed because we had a lot of errors every morning, in particular with calcium, total protein, and albumin.”
That was often a function of the population they were testing, he explains. “Early in the morning the only ones we are testing are hospitalized patients who typically have lower albumin and lower calcium. So we had a lot of false-positives and a lot of false-negatives where we were not catching errors because of the way our population was skewed. We ended up having to split off inpatients and outpatients into separate groups to monitor them separately.”
Laboratories investing in PBRTQC have to decide how large an error should be before it becomes significant, he says. “Depending on how extensive you make your protocol, you can detect minor perturbations in your population that may be analytically significant but probably aren’t going to rise to the level of concern for a clinician. So it becomes a balancing act. We do still get false-positives on occasion, largely when we get new lots of reagents. For example, with calcium our average might shift up to a few tenths of a milligram per deciliter. The way we have our system set up, because of that small shift we can occasionally get some false error detection.”
To prove the value of having real-time QC over conventional QC, it might be useful to find another hospital lab of a similar size, test composition, and volume and then compare results corrections, Dr. Cervinski says. But it can be delicate to make comparisons. “It’s kind of hard. And we don’t have an objective measure yet. We run CAP Surveys, but those are such fixed points in time that unless that error is happening on the day you run it, it’s not going to make a difference.” However, he points to LabCorp’s data, cited by Dr. Katayev, showing that LabCorp’s PBRTQC system reduced the number of samples that needed to be rerun. While he has not formally collected data at Dartmouth, “I know personally that we’ve been able to detect errors that would otherwise go uncaught for many, many hours and would have resulted in many more patient samples being affected.”
The work of setting up the different protocols to launch PBRTQC is worth the result, in Dr. Cervinski’s view. “We’re not going to be able to monitor every test we do with patient-based QC. But there’s tremendous value in being able to monitor our equipment in real time. So patient-based real-time quality control is an enhancement that is of benefit. It is a great tool.”
Anne Paxton is a writer and attorney in Seattle.